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ABSTRACT 


To determine the attitudes and opinions of Naval Personnel in re- 
gard to career incentives, retention, education, conditions of Navy life, 
etc., an annual mail survey is conducted. The fact that response is vol- 
untary introduces a risk of bias in the results due to nonresponse. This 
study examines data consisting of demographic variables on the enlisted 
personnel participating in the 1969 Navy Personnel Survey to determine if 
differences exist between those who responded to the survey and those 
who did not. Additionally, the premise that the more successful Navy men 
respond with a greater frequency than those who are less successful is 
analyzed. An empirical classification schems tor determination of success 


using certain demographic variables is presented. 
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I. INTRODUCTION 


The recent advances in technology still have not precluded the need 
for Navy program managers to obtain meaningful feedback to supplement 
their decision-making processes. For it is these program managers who 
daily make decisions which influence many thousands of Navy personnel 
in regard to job satisfaction and/or career motivation. 

The NAVY PERSONNEL SURVEY series was established so that inform- 
ation might be gathered for use by Navy managers in the evaluation of 
personnel plans and programs and in the formulation of Navy policy. V 


Periodic Navy~wide sample surveys are used to systematically collect 


f 
V 
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attituce and opinion data from Navy personnci. Commenting on the Navy 
Personnel Survey 1969-1 (NPS 69-1), VADM Charles K. Duncan, then Chief 


of Naval Personnel, wrote, "...These surveys are actually conduits for 
the transmission of the attitudes of a large segment of our Navy popula- 
tion. Let us listen to what they are saying and be guided to the maximum 
feasible extent by this fleet and shore feedback." l 

The successful operation of a large-scale sample survey is nota 
simple undertaking. The mailed questionnaire is a common technique of 


surveys because of the economies involved. However, a frequent objec- 


tion to this method of collecting factual information is that it may involve 


i 
Naval Personnel Research and Development Laboraicry WSR 71-3, Report 


of Enlisted Findings Navy Personnel Survey NPS 69-1, sy Claude Braunstein, 
bead, august 1970: 


a 





a large nonresponse rate, and an unknown bias in the assumption that 
those responding are representative of the combined total of respond- 


ents and nonrespondents. 





Il, PROBLEM 


The mailed questionnaire is widely used as a technique for gather- 
ing desired information, especially in the social sciences. This instru- 
ment of research allows one to reach a large group of individuals and 
cover a wide area, both geographically and culturally, for a relatively 
low cost. Other advantages to the mailed questionnaire include elimina- 
tion of interviewer bias, possible gain in validity by assurance of anony-- 
mity, and hopefully, greater consideration by the individual in making his 
responses. 

NPS 69-1 was the seventh survey in the Navy Personnel Survey 
series. Ine survey questicnnaires were mailed directly to 24,900 enlisted 
personnel on 20 June 1969 (officers were also included in the survey but 
were not considered for the purposes of this research). The importance of 
answering the questionnaire was stressed and the men were asked to re- 
turn the self-sealing pre-addressed answer sheet directly to th Naval 
Personnel Research and Development Laboratory within five days of receipt. 
A follow-up letter, together with a duplicate questionnaire, was sent to 
all nonrespondents five weeks after the initial mailing. Returns were 
accepted through 29 August 1969. 

The fact that response was purely voluntary introduced a risk of bias 
in the results due to nonresponse. In general, researchers appear to be in 


agreement that nonrespondents do differ in some ways from the respondents. 





The problem then is to determine if these differences tend to bias the sur- 
vey results, and, unless the nature of this bias is not determined, it is 
then not possible to use the answers of the respondents alone to general- 
ize about the entire population. 

This paper cannot attempt to solve the bias problem explicitly since 
the data analyzed did not include information on how the respondents 
answered the questionnaire on an individual basis nor on how the non- 
Bondents would have replied had they answered the questionnaire. It —- 
does, however, seek to identify demographic variables whose values are 
correlated with frequency of nonresponse. Since the analysis lacked data 
specifically related to response, it was not possible to investigate the 
nature of magnitude of bias. Ideally, this study has created a framework 
under which bias can be studied as well as an identification of those areas 
where factors can be applied to allow for variability in frequency of 
response. 

The major premise considered herein is that successful Naval per- 
sonnel respond with a greater frequency than do those who are less suc- 
cessful. A scheme for classifying personnel into success categories is 
developed to examine the claim. The final section presents two methods 
for measuring bias in mail surveys when there is only one variable under 


consideration in the survey. 





TTI. LITERATURE REVIEW 


Although a review of the literature reveals differing views on the 
validity of mailed questionnaires, the general consensus appears to be 
that people who reply to a questionnaire in many instances are different 
from those who do not reply. Some researchers felt that the mailed ques- 
tionnaire approach is effective only when sampling within a "homogenecus" 
group. Clausen and Ford, in particular, indicated that a small percentage 
of nonresponse may be relatively unimportant if one is working with a 
homogeneous group. However, the literature has been lacking in specific 


guidelines for defining "homogeneous" in the context of response bias. 
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Mie fOiowing constiivics Clausen and Ford's diese, 

"Tf we define a homogeneous group by any 
criteria other than the characteristic which 
is to be estimated and there will still be 
cleavages within the group... 
The crux of the problem of working with a 
homogeneous group is this: How do you 
know they are homogeneous with respect 
to the relevant characteristics (including 
interests) until you have made a study of 
a representative segment of the group?" 


In a similar context Goode and Hatt reported, "The central point... 


necessary to underline is the fact that the mailed questionnaire is not an 


effective research tool for any but a highly select group of respondents. nd 


Ah aiseny J.A., and Ford, R.N., "Controlling Bias in Mail Questicn- 
naires," Joumal of American Statistical Assoc., v. 42, p. 497-511, 1947. 


3Goode, William J., and Hatt, Paul K., Methods in Social Researcn, 
Dae 4. McGraw Hill, 1952. 
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Here too, the concept of a highly select group has gone relatively unde- 
fined. Certainly there exists sufficient reason for believing that the 
sample for the NPS series would constitute such a group. 

A study conducted in the early 1940's by C. F. Reuss found that 
certain marked differences existed between respondents and nonrespondents 
to a mailed questionnaire. Some of Reuss’s conclusions on his respondent 
population will be especially applicable to the respondents of the NPS 
Scies. 

The subjects of the Reuss study were members of the 1936 freshman 
class at the State College of Washington. A wealth of background inform- 
ation was available on all subjects from records in the Registrar's office. 
An analysis of both groups of respondents on the basis of their background 
variables showed that "higher intelligence scores and scholarships, loyal- 
ty or ties to the questionnaire sponsor, and a rural background seem to be 
positively associated with the tendency to respond. 4 Loyalty or ties to 
the sponsor was thought to be a function of length of stay in college. 
Those individuals who had stayed in college for at least three years were 
more likely to answer the questionnaire than those who left after a shorter 
length of time. In addition, more than one-third (37%) of the responding 
group, but less than one-sixth (14.7%) of those not responding had re- 


ceived a degree from the State College. Hence, it was thought that length 


4Reuss, Carl F., "Differences Between Persons Responding and not 
Responding to a Mailed Questionnaire," American Sociological Review, 
v. 8, p. 433-438, 1943. 
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of stay and the influence of the degree suggested a feeling of loyalty to 
the institution which was a factor strongly influencing questionnaire re- 


sponse. In conclusion, Reuss notes, ©... unless a substantial proportion 
of coverage is secured, the returns from the mailed questionnaire cannot 
be assumed to be adequately representative of the universe from which 
they are drawn." Noteworthy was that Reuss did not define what he meant 
by "a substantial proportion," although his study secured a 67 percent 
response rate. 

Since the concept of loyalty or ties with the sponsor will be evident 
in the results of this study, it would be of interest to consider another 
survey that crew similar conclusions. Edgerton, Butt, and Norman con- 
ducted a study of contestants involved in the First Annual! Science Talent 
Search.” They classified the contestants into three classes: Winners, 
Honorable Mentions, and Others. Response to a questionnaire sent out 
by Science Service, the sponsoring agency for the Talent Search, indicated 
that the winning contestants made almost perfect returns, the Honorable 
Mentions group made the next largest percentage of returns, and the 
Others had the lowest percentage of returns. In addition to the loyalty to 
the sponsor concept, the authors felt that an interest in the subject matter 
of the questionnaire (science in general) as evidenced by winners being 


perhaps "more interested," was also a factor. Note also that their 


‘Edgerton, Harold A., Butt, Steuart H., and Norman, RalphD., 
“Objective Differences Among Various Types of Respondents to a Mailed 
Questionnaire," American Sociological Review, v. 12, p. 435-444, 1947. 
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classification of contestants could be considered as a success indicator 
with the more successful individuals (winners) responding with the greater 
frequency. The authors recommended stressing intensive follow-up to mail 
questionnaire research since "... the tendency will be to obtain replies 
from those who have a special interest in the subject under study, or who 
exhibit some characteristics or characteristic different from non-respond- 
ents or from the casual or indifferent respondents." 

A team of ‘oeaie ers who support the claim that there are no differ- 
ences between respondents and nonrespondents are McDonagh and 
Rosemblum. They selected a 10 percent random subsample from subjects 
who completed their questionnaires and from those who did not complete 
them. A team of interviewers was carefully chosen to conduct the field 
research of the two subsamples. Key questions in both the questionnaire 
and interview approaches were identical. They found no statistically 
significant differences between the respondents and nonrespondents. 
Hence, their study suggests that the mailed questionnaire may reveal 
representative responses in spite of partial return from the sample selected. 
The authors concluded: 

",.. there were no significant differences 
between the responses of the mailed 
questionnaire and those of the interviewed 
respondents who had not answered the 
questionnaire. The nonrespondents did 
not seem to be so selective of some vari- 


ables as many behavioral scientists 
assume. The findings of this study imply 
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that researchers should have greater 
confidence in the questionnaires 
method as an initial tool of research." 

Since the data used for this paper consisted primarily of demographic 
variables on the personnel from the survey sample, it was thought that the 
comments of C. R. Bell would be appropriate for this review. d He noted 
a lack by any reseacher to attempt a systematic description of the volun- 
teer. He decided to divide the variables examined for their association 
with volunteer bias into three categories: sociological, psychological, and 
other, which included the "mechanics" of the questionnaire itself. Some 
of the sociological variables included age, ethnic background, marital 
status, occupational status, sex, years at school, etc. Bell's conclusion 
on these variables was that those studied and conclusions drawn are rarely 
the same in any two reports. In certain studies some variables are shown 
to be associated with bias and in others they are shown not to be. In 
addition, some of the variables cited may he factors relevant to physical 
availability at the time of the survey (e.g. young mothers and elderly 
retired persons may deceptively appear to be "volunteers" when merely 
they are those who are rarely not-at-home when the interviewer calls). 

some of the psychological variables used to describe a volunteer 
included better adjusted, more drive, more interest in the topic, lonesorne, 
more articulate, etc. Bell notes that attempts to characterize the volunteer 

SMicDonagh, E.D., and Rosenblum, A.L., "A Comparison of Mailed 


Questionnaires and Subsequent Structured Interviews," Public Opinion 
KOuomuerhe; VV. 29, Dp. 131-136, 1965. 


Bell, C.R., "Psychological Versus Sociological Va’ lables in Studies of 


ec Bias in Surveys ," Journal ofApplied Psychology, v.45, p. 80-85 
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using these types of variables have been hardly more fruitful than with 
sociological variables. The most acute problem seems to be translating 
the findings expressed in personality tests jargon into a meaningful con- 
text outside the laboratory (e.g. it is not easy for the market researcher 
to usefully convert such descriptions of the volunteer as one having "a 
greater self-discipline and tolerance of others"). 

Observing trends in data from repeated mailings has received some 
attention in the literature as a device for learning something about the 
nonrespondents ina mail survey. As an example, consider the survey 
conducted by Huddleston of 3,241 North Carolina fruitgrowers a The aver- 
age number of fruit trees per farm (hereafter referred to as X) was known 
to be 329. However, the results showed some interesting characteristics. 
The first mailing yielded just 300 returns with X = 4560. A second request 
to the remainder of the list yielded 543 returns with X = 382. The third 
and last request yielded 434 returns with X = 340, elhewva lie: of X in every 
case refers only to the farms who responded to that particular request. 
Noteworthy was the X in each wave of returns becomes progressively 
smaller but yet it still overestimates the known true value. The author 
indicated that, on the average, farms having large numbers of fruit trees 
are more willing to respond than farms having smaller numbers of trees. 
He concluded that a farmer's interest in a fruit survey can thus logically be 


expected to be positively correlated with his scale of operations in the 


iabhlie leeksrsevekoneee 


Selector H.F., "Methods used in a survey of orchards," 


Agricultural Economics Research, vol. 2, pp. 126-130, 1950. 
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A mail survey conducted on 1,189 Grade A milk producers in North 
Carolina produced results in which the selectivity of the mail returns was 
in the opposite aircon In this survey it was known that the average 
number of cows per farm (hereafter referred to as Y) for all 1,189 farms 
was 24.27. 

The first mailing yielded 165 returns with ve 23.03. A second 
request received 170 returns with Y = 23.79. The third and final request 
yielded 114 returns with Y = 24.23. As before, the value of Y refers only 
to the farms who responded to that particular request. The bias of the mail 
returns with respect to cows per farm was not as large as the bias in the 
fruit trees survey. However, the interesting aspect of the bias was that 
farms with smaller numbers of cows were the more willing to respond! 
This would appear contradictory to the results from the previous survey 
where a farmer's interest in reporting increased with the scale of his oper- 
ations. The author noted that while there were factors such as scale of 
operations which induced a farmer to report, there were also other factors 
pulling in the opposite direction at the same time. Finkner's conclusion 
was that the amount of work and time required to fill out the questionnaire 
increased as the scale of operations increased and that this apparently 
created negative influences for completing the questionnaire. 

No review of the literature on mail surveys would be complete with- 


out at least some mention of the Literary Diaest fiasco of 1936. During 


renner A.L., "Adjustments for nonresponse bias in a rural mailed 
survey, Agricultural Economics Research, vol. 4, pp. /7-82, 1952. 
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the 1920's and early 1930's the Digest polled millions of citizens with 
postcard ballots and had established a fairly accurate record. For example, 
in 1932 it differed by less than 1 percentage point in predicting the vote 
for Roosevelt. In 1936, however, it made a very large error of 19 percent~ 
age points in predicting Roosevelt's vote. Approximately 20% of the ballots 
mailed out were returned to the Digest. From a mailing of ten million or 
more it received some two million ballots yet this huge mail vote was so 
in error that the poll was disestablished. 

F, F, Stephan noted the following in his review of the Digest's 
dilemma: 


"There is general agreement that this mail-ballot 
method was subject to a serious distortion be- 
cause the better educated and more literate part 
OF Ime -Couleation, as well as those who wors 
higher on the economic scale, tended to return 
their ballots in greater proportion than those who 
were lower in educational and economic status. 
In addition, the Digest obtained the names of 
persons to whom they mailed the ballots from 
automobile registration lists, telephone directories, 
and similar sources. These sources were phe sce 


® ® e 4 uf 
upward in education and economic status." 


maNiosiellen and others, The Pre-election Polis of 1948, p. 10 


Social Science Research Council, New York. 1949. 
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IV. DESCRIPTION OF DATA 


This study utilized data from the NPS 69-1 sample since the men 
participating in that survey were identifiable by service number, hence, 
it was possible to separate the respondents from the nonrespondents. 

The data consisted of various demographic variables that were avail- ~ 
able on each sample member from the Naval Personnel Research and 
Development Laboratory's tape of extracts of the Enlisted Master Tape 
from 1969. These variables and a description of their content are as 
ows. 


Rate Code - a S-character alpha/numeric code which is the 
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equivalent o 

General Classification Test Score (GCT) - a 2-digit numerical 
Navy standard score which indicates an individual's ability to 
understand words and relationships between words, thus imirectly 
measuring reasoning ability. 

Education Level - a l-character alphabetic code indicating 
degree/diploma received. 

TAR/STAR/SC ORE Indicator (TSS) - a l-character alphabetic 
code used to identify personnel serving under, or formerly under, 
the following programs: Training and Administrations of Reserves 


(TAR) 


‘Manual of the Active Duty Enlisted Master Magn tic Tape Record, 
NAVPERS 15,949C, Bureau of Naval Personnel, July, 198. 
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Selective Training and Retention (STAR) 
Selective Conversion and Retention (SCORE) 

Expiration of Active Obligated Service (EAOS) - a 6-digit 
numeric code indicating the date current enlistment expires. 

TERM ENLISTMENT - a l-character numeric code which indi- 
cates the number of years for which an individual is currently 
enlisted. | 

NOENL - a 1l-character numeric code which indicates the 
number of enlistments in which an individual is presently serving. 

Date of Birth - a 6-digit date (last two digits of year, month, 
day) which indicates the birth date of an individual as recorded in 
his enlistment contract. 

Dependents - a 1l-character alpha or numeric code reflecting 
primary dependents. 

Current Enlistment Date - a 6-digit date which indicates when 
an individual commenced his current enlistment, as recorded in his 
enlistment contract. 

BUPERS Activity Code - a 10-digit numeric code identifying 
eect V1 tae 

oea/Shore Code - a l-character numeric code which indicates 
sea or shore classification of an activity. 

Homeport Code - a l-character alpha or numeric code which 
identifies the genera! area of the official home port of a ship, or per- 


manent duty station of an aviation unit, certain siiffs, etc. 
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Race Code - a 1l-digit code indicating an enlisted man's race. 

Invariably in any study involving "rea! world" data there will be in- 
stances of missing information. In particular, the fact that Education Level ~ 
was absent for over one third of the individuals, made this variable of 
minimal value in the analysis. Additionally, about 1,200 values for GCT 
were missing. However, a visual perusal cf tne data on those individuals 
where GCT was not included revealed no particuiar trends or characteristics. 
Hence, it was assumed that the validity of any results based on the use 
of GCT was not affected by Pe ciate those individuals whose GCT was 
missing. 

The data lacked a comparison of a particular respondent's demo- 
graphic variables and how he answered the avestionnaire. Information of 
this type would be necessary if one desired an identification of trends in 
responses from specific strata of the respondent sub-sample. Reference 
13 contains the percentage distribution of enlisted responses by rate, 
enlistment/extensions and marital status. 

As noted previously, 24,900 questionnaires were mailed. The num- 
ber of questionnaires returned to the Laboratory by the Post Office as 
undeliverable was 1,086 leaving 23,814. There were 16,645 question- “ 
naires returned in all. Of that total 1,426 were returned after the cut-off 
date and the editing process marines 1,468 leaving a balance of 13,751 
as "true" respondents. The data indicated 13,684 respondents, thus 67 


respondents were missing for the purposes of this study. 
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If one considers 16,645 of 23,814 as the return rate, then there 


were 7,169 "true" nonrespondents. Of course, 23,814 is a maximum esti- 
mate since there is no way of knowing just how many survey gquestion- 
naires did not reach the intended recipient. Unfortunately, during an 
earlier editing and compilation phase of the returns, the 1,426 returned 
after cut-off, 1,086 returned as undeliverable and 1,468 edited were all 
denoted as nonrespondents. Hence, the data should have included 7169+ 
1426+1086+ 1468 = 11,149 as nonrespondents. However, only 10,46] 
were classified thusly, leaving a total of 688 missing. 

Obviously, there was considerable comtamination of the data avail- 
able for the nonrespondent population. Taking 16645/23814 = 0.699 as 
the response rate for the survey, then one would intuitively expect 
0.699 x 1,086 = 759 of the undeliverable questionnaires to have been re- 
turned. This being the case, then 1,426 + 1,468 + 759 = 3,653 who couid 
have been classified as respondents were, in fact, included in with the 
nonrespondents. Hence, if there are any inherent differences between the 
populations one would expect that these differences would not be as read- 
ily apparent from the contaminated data since the characteristics of the 
nonrespondent population tend to approach those of the respondent popul- 
ation. Thatis, itis anticipated that any visible cleavage between the 
two sub-samples would be further polarized had the nonrespondent data 


not been biased. 
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V. ANALYSIS OF DEMOGRAPHIC VARIABLES 


In any research endeavor when large amounts of data are present, 
it is often prudent to begin analysis by computing simole sample statistics 
and constructing frequency distributions. The first factor examined was 
age. The mean age for nonrespondents was calculated as 25.23 years 
and respondents were found to be slightly less than a year and a half 
older with a mean age of 26.77 years. The sainple variances were 41.64 
and 47.78 respectively. It should be noted though, that averages in 
this situation are not an especially representative statistic due to the 
skewed characteristic of the age distribution cf service personnel. How- 


41 a ee es a he, i * ° s e s 
me fofia ci that distribution it is pecerhble 


ever, without prior kKnowieage of ¢t 
with the aid of the Central Limit Theorem to assume a normal distribution 
for the sample means and perform a test of hypothesis concerning their 
equality. 


Assuming the variances to be equal, the appropriate test statistic 
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where Sp is the pooled estimate of the sample variances, xy and Xx, are 


the two sample means, and ny and n are the respective sample sizes. 


The above statistic is distributed t however, inthe limit as the 


+. = f 
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number of degrees of freedom becomes large, the t dist ibution approaches 
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the normal. Hence, with the value of the test statistic equal to -17.52 
there was sufficient justification to reject the null hypothesis at the 5% 
level that the two means were equal. 

Perhaps more informative than a comparison of mean ages is a fre- 
quency distribution of the ages. Table 1 lists the number of personnel that 
fall within six age categories. Included in parentheses is the correspond- 
ing percentage of the sample. It is then a simple matter to calculate the 
frequency of response within each age category for it is just the number 
responding divided by the sum of the respondents and nonrespondents in 
that category. From the inclusion of this information in the table, it is 
apparent that the likelihood of response increases with age to age 30 and 
remains constant thereafter. As a benchmark for comparison, the response 
rate considered herein will be 0.567 since the data utilized listed 13,684 
respondents and 10,461 nonrespondents for a total of 24,145 cases (see 
Section [V). 

S20 21-25 ZO =o 31-35 36-40 740 
RESPONDENT 6129(25.4)| 2049(8.5)11971(8.2)11120(4.6)} 681(Z. 8) 
nc) 


NON- 3 
RESPONDENT {2971(8-2) | 5210(21.6) | 1219(5.0)}1080(4.5)} 608(2.5)|373(1.5) 
Freq of 0.468 0.540 0.627 0.646 ic 4o mee iG 
Response 


Table 1. Response vs. Age 
Tables of the previous type can be very enlightening when one is con- 
fronted with a mass of data to be analyzed. It allows the researcher to 
systematically group and display the information in a manner in which 
obvious, and often subtle, differences are easily observable. Itis for 


this reason that the remainder of this section is devoted to a display of 





various 2 x n contingency tables reflecting response characteristics where 
n is the number of categories for a particular demographic variable. 
Appendix I contains the results of the Chi-Square Test for two independ- 
ent samples for the above and succeeding tables in this section and for 
two tables from the next section. The hypothesis under test is that the 
two groups do not differ with respect to the demographic variable under 
consideration. 

Table 2 contains the breakdown of response vs. GCT. The scores 
have the following interpretation [Ref. 2] : 


65 and higher - "high" or about 7% of all enlisted personnel 
55-64 - above average or about 24% of all enlisted personnel 
45-54 --average or about 38% of all enlisted personnel 


35-44 - pelow average or about 24% of all enlisted personnel 







34 and below - "low" or about 7% cf ali enlisted personnel 
aa 55-64 45-54 35-44 < 34 
RESPONDENT ae 5212(21.6)] 3782(15.7)| 1568(6.5)| 1033(4.3) 
NON- 
RESPONDENT | 1211(5.0)| 3477(14.4)| 3150(13.0)| 1667(6.9)| 956(4.0) 
ae oe 0.633 0.599 0.484 0.519 
esponse 


Table 2. Response vs. GCT Category 
As a side note, it is apparent, at least from this sample, that today's 
Navymen are more intelligent than in the days when the original GCT dis- 
tribution was devised. Almost twice the number of individuals comprised 
the 65 and higher group as compared to what the theoretical distribution 
had predicted. Likewise, in the above average group there were one-third 


more than expected. As a result, the next two grou.s were low but, 
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interestingly, the "low" group comprising 8.3% of the sample was 
quite close to the anticipated percentage. 

With the exception of the "low" group, note the monotonically de- 
creasing order of the frequencies of response for the GCT categories. As 
mentioned in Section IV, the fact that about one-third of the nonrespond- 
ents could actually be classified as respondents, it would be expected 
that under more ideal conditions not only would the frequencies be greater 
for the higher GCT classes but that monotonicity would be preserved, thus 
establishing an intuitively appealing relationship between GCT and 
response. 

There are six categories under the Sea/Shore code: shore, sea over- 
seas, toured sea duty (non-rotated ships), preferred sea, and preferred 
overseas. Table 3 contains a numerical breakdown tor Sea/Shore code. 


Toured Preferred Preferred 





Shore 9ea Overseas Sea Duty Sea Duty Overseas 
RESPONDENT | 4862(20. 1)|5904(24.5)11243(5.1)1773(3.2) | 275(1. 1)|627(2.6) 
aN 2984(12.4)15159(21.4)| 957(4.0)1869(3.6 Ene) 
Ren | 25 7 57(4.0}1869(3.6)| 156(0.6)1336(1.4) 


Freq. of 0.619 0 503 0.540 0.470 0.638 0.65] 
Response 


Table 3. Response vs. Sea/Shore Code 


The greatest frequency of response was associated with the preferred 
overseas duty group. The next greatest were preferred sea and shore duty 
respectively. As might be expected, overseas duty and sea duty had lower 
response rates, while interestingly, those on toured sea duty responded at 


a rate of only 0.470. 


29 








Table 4 records response as a function of race. The response rates 
of 0.707 and 0.619 for Indian and Mongolian, respectively, should be taken 
with fairly low confidence due to the small sample sizes and the contami- 


nation that was present in the data. ) 


Indian 
Caucasian Negroid (Am) Malayan Mongolian 


RESPONDENT RZ ole, mre) 6 7X om2)e| ZOOL) 29 On ZOO y) 
) 


en), ) 
NON- ; ; ; 
RESPONDENT oF 5031393 SCH Sy) IG), i) Lil espa een (0) 10) 
Freq. of Oro 0.480 O07 On as 0.619 
Response 


Table 4. Response vs. Race 


Table 5 contains the response characteristics when compared to the 


number of enlistments of a sample member. 


1 Z 3 ef S:Or Were 
RESPONDENT Zs 8 39) | 2352\ ce) 
NON- 
ReofONDENT 
Freq. of 
Response 





Table 5. Response vs. Number of Enlistments 


Of note here is that frequency of response increases with the number of 
enlistments until the fourth and then tapers off somewhat. Recall from 
Section ITI the study conducted by C. F. Reuss where a relationship was 
established between the response rate and loyalty to the originating agency. 
Similarly, the longer an individual remains in the Navy, as reflected by 


the number of enlistments, the greater are the ties that the individual 
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has for the Navy and, hence, the greater is the probability of response 
from that individual. 

Finally, the last demographic variable considered in this series is 
the number of dependents. There are 18 categories under this heading: 
none, Wife, wife and 1 child, ..., wife and 8 or more children, 1 depend- 
ent child, ..., 8 or more dependent children. Table 6 contains a condensa- 
of the results into 9 categories. 


Frequency of 
Dependency Status Respondent Nonrespondent Response 


None CUS 2 eo) 5846(24. 2) Ores. 
Wife 225A 3) 1646(6.8) Oe 577 
Wife & 1 child homo on 7) LO) a) O= 605 
Wife & 2 children 1614(6.7) 878(3.6) 0.647 
i wy 3 ‘ 1081(4.5) Bl( ea) 0,677 
" 4 or more 

children SES) 456(1.3) 0.652 
ime nild FOO. 3) 57027) 0.574 
2 children 50(0. 2) 36(0.1) 0.581 
3 or more children 43(0.2) 221021) 0.662 


Table 6. Response vs. Dependency Status 


Zi. 





VI. THE SUCCESS FACTOR 


The primary objective of this study was to examine the premise that 
the more successful Navymen respond with a greater frequency than do 
those who are less successful. If this is the case and additionally it can 
be proven that an individual's responses differ as a function of his success- 
suoupunembership, then the results will be biased in the direction ef the 
proportionality of the group sizes. The first task in answering the above 
was to develop some criterion for success. Ultimately, what one would 
like for analyzing this situation would be a complete record of each indi- 
vidual's advancement-in-rate history. Then, all-Navy averages could be 
used aS a comparison and an individual would bo assigned an index which 
would signifv his "rank," or success factor, within his peer (rate) group. 
The determination is then made that the more successful individuals ad- 
vance more quickly, those of average success advance about at the rate 
of the all~Navy average, and the less successful obtain promotions ata 
slower rate. 

Unfortunately, the data used for this study did not include such ad- 
vancement history. An alternative, however, was to assume that, on the 
average, individuals enlist in the Navy at or near the same age. Then one 
need only compare an individual's age with his paygrade, or rate, to deter- 
mine the individual's advancement, e.g., E-5 at age 23, E-7 at age 27, 
etc. What remains is some determination of how this ccmpares with the 


other Navymen. 
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Before answering that question, the assumption of same age at first 
enlistment requires further consideration. Table 7 contains information on 
the age at enlistment for all the first-term (first enlistment) personnel from 
the sample population. Also included in that table for comparison is data 
(Ref. 14] which gives the percent enlisting at each age for Quarter IV- 

FY 70 (1 March- 30 June 1970). 


ALL-NAVY QTR IV- 


AGE NO. PERCENT FY 70 PERCENTAGES 
17 643 4.60 8.00 
18 3703 26.50 24.50 
19 5147 36.80 24.30 
20 2284 16.30 26.50 
21 870 6.20 8.80 
22 555 4.00 3.40 
23 265 1.90 2.50 
24 i57 0.50 1.10 
25 98 0.70 0.30 
26 32 0.20 9.20 
27 25 0.18 0.07 
28 21 Os 0.03 
29 29 0.20 0.04 

> 30 156 Lea Geet 

13986 


Table 7. Frequency Distribution of Age at First Enlistment 


The majority of the ages appear to be centered around age 19 in both 
cases. More precisely, the mean and the median of the distributions occur 
at 19 years. Therefore, if one had to choose an age which was most cha- 
acteristic of the age at first enlistment, for the sample, 19 years would be 
the logical choice since almost 80% of the other ages fall within one year 
of it. It was felt that this was sufficient faetlsication for assuming that, 


on the average, individuals enlist in the Navy at or near the same age. 
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The next step in assessing the success factor was to establish the 
criterion for success itself. This was accomplished by stratifying the 
sample by age (17, 18, ..., 2.50) and pay grade (E-1 through E-9) and dis- 
playing the number of personnel within each category in a 34x9 matrix. 
This was done for both the entire population and careerists alone (more 
than one enlistment). See Appendices A and B, respectively. 

Each age-pay grade combination comprises a "cell" and there are 396 
cells in all. Obviously, not all cells are non-empty since there is very 
little likelihood that a 19 or 20 year old has attained the pay grades of E~6 
through E-9. Similarly, the chances are low that an individual of 35 years 
of age or older is an E-1 through E-3 unless, of course, as a result of dis- 
@aplimery action. 

The technique used for determining success is empirical in nature. 

It utilizes a notion appealed to by many stétisticians; that is, data relating 
to human performance is in many ways very nearly normally distributed. 

The majority of the population falls in the average category which is with- 
in one standard deviation of the mean, while the above and below average 
people occupy the extremes of the normal "tails" to the right and left, 
respectively. Similarly, we can consider three categories of success: 

the more successful group, the average success group, and the less suc- 
cesstul group. 

To see how the empirical method for determining success works, 


refer to Table 8 which is a partial extract of the matrix from Appendix A. 
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PAY GRADE 


17 0 1 0 0 0 
18 1 quD 1 12 0 
19 2 370 cis 9 
20 8 402 861 14) 177 
21 A 148 663 1630> 598 
22 1 50 388 8272 1510 
23 1 7 174 854 Coz) 
24 1 7 93 414 6 


Table 8. Extract of Age-Pay Grade Matrix 


The first step of the procedure is to scan each row, or age, and circle the 
largest number in that row as illustrated in Table 8. After the rows have 


7 * : ib eireste a aa — ~~ nen A) ~ 1 on . YSU vteo eC le -~s4s07 sO ee a 
Beem SsCalneu the Second Step is tO SCGN CAC: Solumn, St Peay Sracien ages 


¢ 


circle only the number immediately preceding and the number immediately 
following the number(s) previously circled from the row scan. Table 9 ccon- 


tains the results of steps l and 2. 


PAY GRADE 

AGE Eom E-2 ae. 
17 0 => 0 
18 1 C114 C3 
19 2 G70) £381) 
20 8 402 ~\_c86l> 
zi 4 148 663 
22 1 50 388 
23 1 17 174 
24 1 7 93 





Table 9. Results of Applying Steps 1 and 2 of Empirics 1 Method 
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The final step is optional and consists of sketching a line (as illus- 
trated in Table 9) around the circled items to further differentiate between 
the groups. The result is three distinct groups: the circled items repre- 
senting the personnel of average success, and the groups to the right and 
left representing the more successful and less successful personnel, 
respectively. Since the technique is on the order of an "eyeball" routine, 
the procedures for grouping should not be particularly binding thus allow- 
ing the user to modify the circling steps when it appears there are one or 
more numbers especially close to the value of the largest in that row. 

Admittedly in the case for all personnel the procedure created a par- 
ticularly massive average group for the 17 through 20 and E-1 through E-5 
cells. However, this feature can be considered desirable since the 
scheme's primary assumption was a common age at enlistment and a year 
or two difference at that age in the subordinate pay grades could lead to 
considerable classification errors. Note, though, that as age increases 
this year or two difference becomes relatively less critical and the tech- 
nique approaches "stability" with more or less balanced ratios in each 
success group. It is better to be on the conservative side when the pro- 
cedure is particularly susceptible to error. 

Appendices C and D contain the completed success classification 
groupings for all personnel and careerists, respectively. 

The next step in analyzing the success factor was to construct 
matrices similar to those in Appendices A and B, but instead of totals the 


elements were composed of the frequencies of response or each cell 


Cr, 





(i.e., the number of respondents in the cell divided by the total number of 
personnel in the cell). Appendices E and F contain the desired response 
ita . Note the asterisks designate an empty cell to differentiate it from 
a cell with a true zero response rate. 

As might be expected, the final phase of this process was to obtain 
the composite response rates for each success group. Tables 10 and ll 


contain the results for the entire population and careerists, respectively. 


RESPOND- NONRE- RESPONSE 
ENTS SPONDENTS TOA RATE 
MORE SUCCESSFUL 1989 922 2911 0.684 
AVERAGE SUCCESS 9480 7071 16551 0.573 
LESS SUCCESSFUL 2215 2468 4683 0.473 


Table 10. Response Rates of Success Groups for All Personnel 


RESPOND- NONRE- MES oe SL 
ENTS SPONDENTS TOTAL RATE 
MORE SUCCESSFUL 1462 S25 ISS Oo7 
AVERAGE SUCCESS 4937 Zoo 6266 0.644 
LESS SUCCESSFUL oo oe 1908 Oya la 


Table 11. Response Rates of Success Groups for Careerists 


In Section V we observed that personnel with more than one enlist- 
ment responded more frequently than did first-term personnel, so the higher 
response rates for careerists in Table 1] were expected. The above results 
clearly indicate that there exists reasonable evidence to support the claim 
that successful personnel respond more frequently than do those who are 


not as successful. 
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VII. FURTHER CONSIDERATIONS 


To analyze the success factor the previous section dealt with a two- 
way stratification of the survey sample on the basis of age and pay grade. 

It was decided to extend this concept by including an additional dimension 
of stratification -- the general classification test score (GCT). 

As noted in Section V, there are five GCT categories. For economy 
and ease of display the low and below average categories were condensed 
into a low group, and similarly, the high and above average categories com- 
prise the high group. Appendix G contains the frequency distributions re- 
sulting from this three-way stratification for all personnel. 

Table 12 contains a numerical analysis for the number of personnel 


within each category and the corresponding resvonse rates. 


peeCLSos RESPOND- NONRE- RESPONSE 
GROUP GCT ENS SPOONER Mom Ol. RATE 
MORE LOW 123 19 | ZOZ Ceas 
SUCCESSPUL AVERAGE 1408 650 2061 0.683 
HIGH O76 154 530 0.709 
LOW 1094 1124 DIAS 0.493 
AVERAGE AVERAGE 6455 73 11198 0.5576 
HIGH Areal oy 23730 0625 
LESS LOW 654 864 1518 Gres 
AVERAGE Piet 711 2362 0.479 
aco eee hut HIGH Zz 180 Ao 02583 
Table 12. Response Rates Resulting from Age-Pay Grade-GCT 
Stratification 


The results of Table 12 indicate that there exists a more than casual 
relationship between GCT and success category, as might be expected. 


Note that the response rates exhibit a symmetric-like nature between and 
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within success groups. Especially appealing in these results is the fact 
that the greatest response rate, 0.709, is associated with the high GCT- 
successful group, while the smallest response rate, 0.431, is from the 
low GCT-less successful group. 

Having obtained some interesting results with GCT, it was decided 
to consider an individual's occupational specialty or rate code instead of 
GCT as another alternative for the third dimension of stratification. 

BUPERS Report 1080-14 contains the 5-character alpha/numeric equi- 
valency of the rating abbreviations which is used for data processing pur-~ 
poses, (e.g., GM3=02004, GMGC=06041, YNSN=17005, MMCS=3700J). 
The 1080-14 also contains a convenient grouping of the ratings into 12 
categories of similarly skilled ratings. Appendix H lists the 12 groups and 
their respective ratings. Table 13 contains a condensation of the results 
from this stratification scheme. 


RESPONSE RATE FOR 


MORE LESS AGGREGATE GROUP 
@.OUP TOTAL SUCCESSFUL AVERAGE SUCCESSFUL RESPONSE RATE 
J oo OAS: Opa 0.543 ios 
M3 oS 0.705 OO90 Chega 6) OOF 
III 949 OTe 0-703 07 1 ORG GZ 
IV 67 0.769 Oro 0.444 UzOZ7 
V 4702 O57 Oe exsill 0.486 Oresi3 
VI P36 0.800 0.475 0.407 0.442 
VII 3264 0.706 eae! Ore 2 0.564 
VIII 2036 Ure U2369 Ore OZ 0.337 
IX Seo ee? WIZ Qasb.'S O55 52 Oo 
Xx 2099 0.784 0.624 Ue o26 0.634 
x! 104 Ee oy O25: 0.480 O02 S 
All 466 Reo Joy 0 Ol eles: Ooo 2 


Table 13. Results from Age-Pay Grade-Rating Stratification 


_ 
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The relatively low frequency of TOS from Group VI is under- 
standable since it is composed predominately of Seaman and Seaman 
Apprentice, personnel usually in the first enlistment. Recall from Table $ 
that the overall frequency of response for all first-term personnel was 
0.515, the lowest for the five categories considered. Of particular inter- 
est, however, was the 0.337 response rate associated with Group VIII 
(construction), lowest of all 12 groups. The information at hand lacked 
an explanation for the reason behind this unusually low response rate. 
However, recall from Section III the results of the survey conducted on 
milk producers in North Carolina. The feeling of the survey originators 
was that the amount of work required to fill out the questionnaire increased 
and the amount of available free time decreased as the scale of Pe 
increases. It would be interesting to see a comparison ci the work loads 
of the construction ratings as compared to some of the other ratings in the 
Navy. 

As a final consideration in the analysis of the NPS 69-1 demographic 
data the effect of time remaining in the Navy and motivation to respond to 
the survey was examined. First-term personnel with an EAOS date of be- 
tween 6 months and one year from the survey date responded at the rate of 
0.465. Those with 3 to 6 months before release date had a frequency of 
response of 0.492, and those who had less than 3 months remaining re- 
sponded at the rate of 0.479. Hence, the response rate remained more or 


less constant as the time until release from active duty decreased. 
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VIII. MEASURING BIAS IN MAIL SURVEYS 


In general, sampling errors are those errors attributed to the fact 
that only a portion of the universe is selected for study, instead of every 
unit as is done in a complete census. All errors other than sampling errors 
are called nonsampling errors. Included in this nonsampling category are 
errors made by respondents in reporting data, errors made by interviewers 
in recording data, computational errors made in processing the data and, 
in particular, nonresponse in mail surveys. Itis possible that some of 
these errors may be compensating and average out. However, many of 
them are not random in nature and they may well lead to detrimental biases 
fe The fina: results. 

Attempting to completely eliminate the nonsampling errors from a mail 
survey would incur such prohibitive costs that the technique would lose one 
of its principal features -- economy. Perhaps a more realistic approach is 
to eliminate as many of the nonsampling errors as possible and measure 
the effects of the remainder. 

Although a survey may involve tireless efforts by the originators in 
designing the questionnaire and drawing a valid sample from the population 
to be surveyed, the return seldom approaches 100 percent completeness. 
Various devices for dealing with the bias caused by nonresponse have been 
described in the literature. Two, in particular, will be discussed here for 


their applicability to the NPS series. They are observation of trends 
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in data from repeated mailings and a cost trade-off model for double 
sampling. 

The procedure of observing trends from repeated mailings is a parti- 
cularly useful device for learning something about the nonrespondents in 
a mail survey (recall from Section III the case of the North Carolina fruit 
growers). As a hypothetical example, suppose one of the objectives of a 
questionnaire is to find the percentage of personnel in favor of a new policy, 
say, a uniform change for enlisted personnel. 

From a mailing list of 5,000 the first wave yielded 2,000 returns, 
representing 40 percent of the total. A second request to the remainder of 
the list yielded 1,000 returns, representing 20 percent of the original total 
list. The third and last request, sent to the individuals who still had not 
responded, yielded 500 returns, representing 10 percent of the original 
list. The results are shown in Table 14. The percentage of individuals 
desiring the uniform change refers only to those who responded to that 


particular request. 


Returned 
Questionnaires Percent Desiring 
Mailing Number Percent Uniform Change 
1 2000 40 78 
2 1000 20 om 
3 500 10 47 


Table 14. Results of Hypothetical Survey on Uniform Change 


Table 14 indicates that the portion of individuals desiring the uni- 


form change in each wave of returns becomes progressively smaller. An 
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examination of the cumulative results in Table 15 indicates that using just 
the returns from the first mailing would have overestimated the desirability 


for a uniform change obtained from the 3 mailings by about 10 percent. 


Mailing Percent Return % in Favor 
1 40 78 
1+2 60 ae 
14+24+3 70 68.7 


Table 15. Cumulative Results from Hypothetical Survey 


However, 68.7 percent is not necessarily the true portion of those in 
favor. If the trend can be extrapolated, it should be possible to estimate 
the percent in favor of the change corresponding to a cumulative return of 
100 percent. Figure 1 depicts a rough graphical solution to the regression 
probiem (the graphical solution was used for illustrative purposes only -- 
analytical techniques are recommended in actual practice.) In this example 
the percent in favor of the change corresponding to a 100 percent return 


would be about 55 percent. 





100 


Percent 75 
in favor 


20 


29 






meee 





Zo 50 co ILOCr 
Percentage return 


Figure l. 
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A word of caution, however. The literature on regression analysis 
is careful to warn the reader on the hazards of extrapolating the regression 
curve "sufficiently" beyond the range of data for which it was constructed. 
Nevertheless, one could use 55 percent as a lower bound and consider 
68.7 percent as the upper bound. Then the estimate of the percent in favor 
of the change corresponding to a 100 percent return would fall somewhere 
in between those values. At this point it would be nice to have some 
analytical procedure to establish the standard l1- a percent confidence 
interval for the range of values. To this writer's knowledge, however, no 
such procedures presently exist. 

Another approach in dealing with nonresponse involves the use of 
double sampling where a sampie of the nonrespondents are actually inter-- 
viewed. The problem is to determine the number of mail questionnaires to 
be sent out and the number of personal interviews to conduct in following 
up nenresponse to the mail questionnaire in order to attain the required 
precision at a minimum cost. The precision referred to is the maximum 
tolerable standard error of the estimate of the variable or characteristic 
under consideration. 

Assume that the cost equation is given by 7 

C= Gon sae oe ile aera 


1a Be 


where C = cost per questionnaire of mailing 


SEG ene M.H., Hurwitz, W.N., and Madow, W.G., Sample 
ourvey Methods and Theory, vol. 1, p. 474, John Wiley and Sons, Inc., 
oS 3. 
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C. = cost per questionnaire of processing returned 


questionnaires 

C, = cost per questionnaire for interview and 
processing 

n = number of questionnaires sent out 

ny = number of respondents 
= (1/k)n, = number of interviews to be conducted 


where n, is the number of nonrespondents 


According to the principle of optimum allocation, it can be shown that the 


optimum values for n and k can be computed from the following formulas 





= pang? [1+ ( ail IP, | 


lc P Bi 


‘c t Cy 1 
=§ ——— i 
C +C.P-_} 
[So l qi 
where N= size of population to be sampled 
2 ae : 
S = variance estimate of variable under consideration 
Oa rate of nonresponse to mailed questionnaire 
poe ee 


1 2 
E = standard error to be tolerated of the variable 
Note, the results obtained above assume that the variances of re- 
sponses for both respondents and nonrespondents are equal. This is 
usually done in practice since an a priori Knowledge of these values is 


oe PEOOmTsegiven ini Olen Ci el tewocee .« 5 Of -ret,- 7. 
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difficult to estimate. Also, a response rate, Pye must be supplied before 
the optimum n and k can be computed. This can usually be estimated 
with a fair degree of accuracy from results of similar previous surveys. 


Having thus solved for n and k, the number of interviews to be conducted 


is then given by n,/k. 
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IX. SUMMARY AND CONCLUSIONS 


In this paper data consisting of demographic variables on the enlisted 
personnel participating in the 1969 Navy Personnel Survey was analyzed to 
determine if differences exist between those who responded to the survey 
and those who did not respond. In particular, the claim that the more 
successful Navymen respond with a greater frequency than do those who 
are less successful was examined. An empirical classification scheme for 
determination of success was presented. The scheme stratified the survey 
population according to age and pay grade. The addition of GCT score and 
rating as third dimensions of stratification was also considered. Addition- 
aliy, two procedures for measuring bias in mail surveys were discussed. 

In conclusion then, this paper has attempted to identify those demo- 
graphic variables that were correlated with frequency of response. Of the 
six variables considered in Section V, all when viewed separately were 
found to be characteristics on which the respondents and nonrespondents 
differed significantly. Specifically, an individual's age, general classi- 
fication test score, type of duty, race, number of enlistments, and depend- 
ency status were all factors that influenced the response rate. 

The success of an individual was determined to be another variable 
affecting the frequency of response. Three classes of success were con- 
sidered: more successful, average success, and less successful. The 


analysis indicated that the respondents and nonrespon‘’ents differed 
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significantly with respect to success classification. In particular, the 
response rate increased with a corresponding increase in the degree of 


success. 
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APPENDIX A 


AGE-PAY GRADE FREQUENCY DISTRIBUTION FOR ALL PERSONNEL 
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APPENDIX B 
AGE-PAY GRADE FREQUENCY DISTRIBUTION FOR CAREERISTS 


AGE PAY GRADE 

E-l1 £—-2 £€-3 €-4 E-5 ©-6 —E-7 £-8 

17 0 0 0 0 0 0 0 0 
18 0 0 0 0 0 0 0 0 
1S 0 0 0 0 0 0 0 0 
20 0 0 0 10 isl 0 0 0 
Zi 0 0 2 5: 6C 1 0 0 
Ze C 3 5 85 208 26 0 0 
23 0 1 g 66 219 57 0 0 
24 0 C 5 57 247 #129 0 0 
25 0 1 5 53 257 #192 2 0 
2€ C C iv 46 268 307 9 0 
Ze 0 0 6 38 187 297 26 0 
28 1 i 2 21 143 268 47 0 
29 1 1 2 18 128 298 70 O 
30 0 0 3 14 107 323 108 7 
21 6 0 7 17 123 387 #181 13 
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APPENDIX H 


GROUPING OF NAVAL ENLISTED RATINGS 
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APPENDIX I 


CHI-SQUARE TESTS OF INDEPENDENCE 


Ee .° there is no difference between the respondent and nonrespondent 
groups with respect to the demographic variable under consideration 
in Labletivwi—alomeweo 10, Vt 

fi. : the respondent and nonrespondent groups differ with respect to the 


demographic variable 


2 
ene 
Test Statistic: (0, ij) 
Yr k E 
oe eS ij 
is" 
. ew 
where 0. = observed number of cases categorized ini row 
ah : 
of j column 
ES = number of cases expected under Hy to be 


noe _th 
categorized ini row ofj column 


Ff = the nuimoer of rows 


k = the number of columns 
Computed Tabulated / 

Table Demographic Variable x 2 ae —(r— 1) (k=) -Ol,di  “Conclucion 
1 Age Bi Omce 5 Loe Reject H) at Omen 
2. Cer ZW 4 Loa " 

3 Sea/Shore Code TOL 5 Sell : 
4 Race 41.08 a Voe5 i 
S #Enlistments ‘407.15 4 Loe 7 
6 Dependency OOS 8 20 a) 
10 Success (All) CKB ols 2 9.21 4 
1] Success (Careerists) alleles 2 9.21 " 
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